Docker containers#

Course link: https://carpentries-incubator.github.io/docker-introduction/
Video: https://www.youtube.com/watch?v=HelrQnm3v4g

Containers are independent OS-level virtualization to deliver software in packages. They ensure: * Standardisation * Portability * Reliability * Reproducibility

It avoids package/ software conflicts. Can use multiple versions of same package simultaneously. The main program you want to use likely depends on many, many, different other programs (including the operating system!), creating a very complex, and often fragile system. One change or missing piece may stop the whole thing from working or break something that was already running. It’s no surprise that this situation is sometimes informally termed “dependency hell”.

For example, in conda the base environment often becomes a dependency hell.

What is a Container? What is Docker?#

Docker is a tool that allows you to build what are called “containers”. ### A situation * Software A works on Ubuntu * Software B works on Arch Linux

We want another filesystem that we could use to chain together both the software in a “pipeline”. Containers can make it possible with something called a “container host”.

Virtualisation#

Container are virtual computers on host computer. A container can be considered as a lightweight virtual machine. Underneath the container is usually the Linux kernel with some filesystem.

“container image” is a recipe or a template for a container.

Container is – a self-contained, complete, separate computer filesystem. * documentation – there is a clear record of what software and software dependencies were used, from bottom to top. * portability – the container can be used on any computer that has Docker installed – it doesn’t matter whether the computer is Mac, Windows or Linux-based. * reproducibility – you can use the exact same software and environment on your computer and on other resources (like a large-scale computing cluster). * configurability – containers can be sized to take advantage of more resources (memory, CPU, etc.) on large systems (clusters) or less, depending on the circumstances.

Potential applications#

Using containers solely on your own computer to use a specific software tool or to test out a tool (possibly to avoid a difficult and complex installation process, to save your time or to avoid dependency hell).
Creating a Dockerfile that generates a container image with software that you specify installed, then sharing a container image generated using this Dockerfile with your collaborators for use on their computers or a remote computing resource (e.g. cloud-based or HPC system).
Archiving the container images so you can repeat analysis/modelling using the same software and configuration in the future – capturing your workflow.

Docker command line#

Install Docker using these instructions: https://carpentries-incubator.github.io/docker-introduction/setup.html
Create an account here: https://hub.docker.com/
Download this intro file: https://carpentries-incubator.github.io/docker-introduction/files/docker-intro.zip
For windows: https://docs.docker.com/docker-for-windows/install/
For Linux: https://docs.docker.com/install/linux/docker-ce/ubuntu/

Install using the repository#

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg –dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo “deb [arch=$(dpkg –print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable” | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo docker run hello-world

If you are getting error for permission

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Then grant a permission using sudo chmod 666 /var/run/docker.sock

You should be able to run this to see Docker Client and Docker Server.

(base) hell@Dell-Precision-T1600:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:42 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Copy paste#

You can save the version name in a text file.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker --version
Docker version 20.10.12, build e91ed57
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker --version>>version.txt

Docker is working correctly ?#

You will see contents in the directory.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

if you get Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? then something is wrong.

Getting help#

General help: docker --help
Command specific: docker COMMAND --help

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container --help

Usage:  docker container COMMAND

Manage containers

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  inspect     Display detailed information on one or more containers
  kill        Kill one or more running containers
  logs        Fetch the logs of a container
  ls          List containers
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  prune       Remove all stopped containers
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  run         Run a command in a new container
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  wait        Block until one or more containers stop, then print their exit codes

Run 'docker container COMMAND --help' for more information on a command.

Docker Command Line Interface (CLI) syntax#

docker [command] [subcommand] [additional options]

Let us start with prebuilt images, then we will create our own images. # Downloading docker image The docker image command is used to interact with Docker container images.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
hello-world   latest    feb5d9fea6a5   4 months ago   13.3kB

Download image using pull#

docker image pull image_name

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
Digest: sha256:97a379f4f88575512824f3b352bc03cd75e239179eea0fecc38e597b2209f49a
Status: Image is up to date for hello-world:latest
docker.io/library/hello-world:latest
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
hello-world   latest    feb5d9fea6a5   4 months ago   13.3kB

The Hello World image is available here https://hub.docker.com/_/hello-world. We can download the image using docker image pull image_name. # Running a container Use command docker container run container_name.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
hello-world   latest    feb5d9fea6a5   4 months ago   13.3kB
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Downloading alpine image#

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image pull alpine
Using default tag: latest
latest: Pulling from library/alpine
59bf1c3509f3: Pull complete
Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest

The container needs extra command.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run alpine cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

Print Hello world using alpine image#

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run alpine echo 'hello world'
hello world

Running containers interactively#

We wanted to keep the container running so we could log into it and test drive more commands? The way to do this is by adding the interactive flags -i and -t (usually combined as -it) to the docker container run command and provide a shell (bash,sh, etc.) as our command. The alpine Docker container image doesn’t include bash so we need to use sh.

Technically, the interactive flag is just -i – the extra -t (combined as -it above) is the “pseudo-TTY” option, a fancy term that means a text interface. This allows you to connect to a shell, like sh, using a command line. Since you usually want to have a command line when running interactively, it makes sense to use the two together. The prompt would change to this.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run -it alpine sh
/ #

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run -it alpine sh
/ # pwd
/
/ # ls
bin    etc    lib    mnt    proc   run    srv    tmp    var
dev    home   media  opt    root   sbin   sys    usr
/ # whoami
root
/ # echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ # exit
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$

Installing Ubuntu#

Find different versions of Ubuntu here https://hub.docker.com/search?q=ubnutu%20&type=image&sort=updated_at&order=desc.

Installing one of them:

(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker pull xufuhe/ubnutu
Using default tag: latest
latest: Pulling from xufuhe/ubnutu
d5c6f90da05d: Pull complete
1300883d87d5: Pull complete
c220aa3cfc1b: Pull complete
2e9398f099dc: Pull complete
dc27a084064f: Pull complete
Digest: sha256:34471448724419596ca4e890496d375801de21b0e67b81a77fd6155ce001edad
Status: Downloaded newer image for xufuhe/ubnutu:latest
docker.io/xufuhe/ubnutu:latest
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker image ls
REPOSITORY      TAG       IMAGE ID       CREATED        SIZE
alpine          latest    c059bfaa849c   2 months ago   5.59MB
hello-world     latest    feb5d9fea6a5   4 months ago   13.3kB
xufuhe/ubnutu   latest    ccc7a11d65b1   4 years ago    120MB
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$ docker container run -it xufuhe/ubnutu sh
# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
# apt-get --help
apt 1.2.24 (amd64)
Usage: apt-get [options] command
       apt-get [options] install|remove pkg1 [pkg2 ...]
       apt-get [options] source pkg1 [pkg2 ...]

apt-get is a command line interface for retrieval of packages
and information about them from authenticated sources and
for installation, upgrade and removal of packages together
with their dependencies.

Most used commands:
  update - Retrieve new lists of packages
  upgrade - Perform an upgrade
  install - Install new packages (pkg is libc6 not libc6.deb)
  remove - Remove packages
  purge - Remove packages and config files
  autoremove - Remove automatically all unused packages
  dist-upgrade - Distribution upgrade, see apt-get(8)
  dselect-upgrade - Follow dselect selections
  build-dep - Configure build-dependencies for source packages
  clean - Erase downloaded archive files
  autoclean - Erase old downloaded archive files
  check - Verify that there are no broken dependencies
  source - Download source archives
  download - Download the binary package into the current directory
  changelog - Download and display the changelog for the given package

See apt-get(8) for more information about the available commands.
Configuration options and syntax is detailed in apt.conf(5).
Information about how to configure sources can be found in sources.list(5).
Package and version choices can be expressed via apt_preferences(5).
Security details are available in apt-secure(8).
                                        This APT has Super Cow Powers.
# exit
(base) hell@Dell-Precision-T1600:~/Desktop/Docker$

docker container run xufuhe/ubuntu cat /etc/os-release

docker container run xufuhe/ubuntu apt-get --help

--rm: this option guarantees that any running container is completely removed from your computer after the container is stopped.
--name=: By default, Docker assigns a random name and ID number to each container instance that you run on your computer. If you want to be able to more easily refer to a specific running container, you can assign it a name using this option.

Summary#

The docker image pull command downloads Docker container images from the internet.
The docker image ls command lists Docker container images that are (now) on your computer.
The docker container run command creates running containers from container images and can run commands inside them.
When using the docker container run command, a container can run a default action (if it has one), a user specified action, or a shell to be used interactively.

Cleaning up containers#

Removing container#

The command docker ps serves the same purpose as docker container ls.

Use docker ps --all to see all running or tracked containers.

(base) hell@Dell-Precision-T1600:~$ docker ps --all
CONTAINER ID   IMAGE           COMMAND                 CREATED       STATUS                     PORTS     NAMES
1b5e9aa58feb   xufuhe/ubnutu   "sh"                    5 hours ago   Exited (0) 5 hours ago               nervous_shaw
21333093a18b   alpine          "sh"                    6 hours ago   Exited (0) 6 hours ago               epic_jang
644d45eaf5f6   alpine          "sh"                    6 hours ago   Exited (130) 6 hours ago             vigorous_hypatia
77194213259d   alpine          "echo 'hello world'"    6 hours ago   Exited (0) 6 hours ago               vigorous_cray
184fbf917972   alpine          "cat /etc/os-release"   6 hours ago   Exited (0) 6 hours ago               dreamy_sinoussi
8045747df6d4   alpine          "/bin/sh"               6 hours ago   Exited (0) 6 hours ago               amazing_hawking
009e47dc71a8   hello-world     "/hello"                6 hours ago   Exited (0) 6 hours ago               serene_murdock
ddd7faafe140   hello-world     "/hello"                7 hours ago   Exited (0) 7 hours ago               peaceful_liskov

Remove a container#

docker container rm CONTAINER_ID

(base) hell@Dell-Precision-T1600:~$ docker container rm ddd7faafe140
ddd7faafe140

Remove all containers#

docker container prune

(base) hell@Dell-Precision-T1600:~$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
1b5e9aa58feb4fc84d1330f2447acd2d109b316d8cc74121bb1fe65811754ba9
21333093a18b17e6f9f3b8b3c715444188200aaae10e885e74495e81451554fa
644d45eaf5f650044bc70082a62502b22158757a5d9308a4296053e6013111cb
77194213259dd34ed51fefa048ab8b861e4138c897cce43c98da097154ae1bdd
184fbf917972bafcf475d486bff80bd1c44d893bc32075f4931a04b87492f896
8045747df6d48e23ed3baf682f24967abd08df9f5c99d8a7341695cae7ab8ac0
009e47dc71a89f981cda1a23cdf7fcb7e42e56c2db134db60847424029bd5874

Total reclaimed space: 55B
(base) hell@Dell-Precision-T1600:~$ docker ps --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Removing images#

docker image rm image_name

(base) hell@Dell-Precision-T1600:~$ docker image ls
REPOSITORY      TAG       IMAGE ID       CREATED        SIZE
alpine          latest    c059bfaa849c   2 months ago   5.59MB
hello-world     latest    feb5d9fea6a5   4 months ago   13.3kB
xufuhe/ubnutu   latest    ccc7a11d65b1   4 years ago    120MB
(base) hell@Dell-Precision-T1600:~$ docker image rm xufuhe/ubnutu:latest
Untagged: xufuhe/ubnutu:latest
Untagged: xufuhe/ubnutu@sha256:34471448724419596ca4e890496d375801de21b0e67b81a77fd6155ce001edad
Deleted: sha256:ccc7a11d65b1b5874b65adb4b2387034582d08d65ac1817ebc5fb9be1baa5f88
Deleted: sha256:cb5450c7bb149c39829e9ae4a83540c701196754746e547d9439d9cc59afe798
Deleted: sha256:364dc483ed8e64e16064dc1ecf3c4a8de82fe7f8ed757978f8b0f9df125d67b3
Deleted: sha256:4f10a8fd56139304ad81be75a6ac056b526236496f8c06b494566010942d8d32
Deleted: sha256:508ceb742ac26b43bdda819674a5f1d33f7b64c1708e123a33e066cb147e2841
Deleted: sha256:8aa4fcad5eeb286fe9696898d988dc85503c6392d1a2bd9023911fb0d6d27081
(base) hell@Dell-Precision-T1600:~$ docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
alpine        latest    c059bfaa849c   2 months ago   5.59MB
hello-world   latest    feb5d9fea6a5   4 months ago   13.3kB

The reason that there are a few lines of output, is that a given container image may have been formed by merging multiple underlying layers. Any layers that are used by multiple Docker container images will only be stored once.

Finding Containers on Docker Hub#

The Docker Hub is an online repository of container images, a vast number of which are publicly available.

There are official images endorsed by Docker itself. They are reliable (no malware) and stable.

Exploring Container image versions#

A single Docker Hub page can have many different versions of container images, based on the version of the software inside. hese versions are indicated by “tags”. When referring to the specific version of a container image by its tag, you use a colon : like this:

CONTAINER_IMAGE_NAME:TAG

For Python 3.8: docker image pull python:3.8
For Python 3.6: docker image pull python:3.6

Other image providers#

ContinuumIO (developer of Anaconda): https://hub.docker.com/u/continuumio
rocker for R langauge: https://hub.docker.com/u/rocker

To download Individually managed containers:

OWNER/CONTAINER_IMAGE_NAME:TAG

Docker Hub is a repository#

So, to acquire an image from a repository with specific version.

OWNER/REPOSITORY:TAG

# Creating your own Container images * If you want custom images for your task then create your own docker. * If you want to run a code written in older version of let's say Tensorflow. * Share you workflow with someone else

Experimenting with `alphine` container#

Check if apt package is installed using which apt To install packages using apt package manager we need to install it first or choose a container that has apt installed. Remember, apt is a newer package manager and apt-get is older one.

Only for `apk` package manager…#

Do not follows this.

The Alpine version of Linux has a installation tool called apk. Type apk to see help.

To install Python3 using apk: apk add --update python3 py3-pip python3-dev

/ # python3 --version
Python 3.9.7

Now we can install any python package of our choice such as ipython.

/ # pip install ipython
Collecting ipython
  Downloading ipython-8.0.1-py3-none-any.whl (747 kB)
     |████████████████████████████████| 747 kB 7.1 MB/s
Collecting decorator
  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting black
  Downloading black-22.1.0-py3-none-any.whl (160 kB)
     |████████████████████████████████| 160 kB 11.6 MB/s
Collecting jedi>=0.16
  Downloading jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 11.8 MB/s
Requirement already satisfied: setuptools>=18.5 in /usr/lib/python3.9/site-packages (from ipython) (52.0.0)

Once we exit, these changes are not saved to a new container image by default. There is a command that will “snapshot” our changes, but building container images this way is not easily reproducible. Instead, we’re going to take what we’ve learned from this interactive installation and create our container image from a reproducible recipe, known as a Dockerfile. # Put installation instructions in a Dockerfile

A Dockerfile is a plain text file with keywords and commands that can be used to create a new container image.

Navigate to the directory we just downloaded. This is content of Dockerfile.

(base) hell@Dell-Precision-T1600:~/Desktop/Docker/docker-intro/basic$ ls
Dockerfile
(base) hell@Dell-Precision-T1600:~/Desktop/Docker/docker-intro/basic$ cat Dockerfile
FROM <EXISTING IMAGE>
RUN <INSTALL CMDS FROM SHELL>
RUN <INSTALL CMDS FROM SHELL>
CMD <CMD TO RUN BY DEFUALT>

Let’s break this file down: * The first line, FROM, indicates which container image we’re starting with. It is the “base” container image we are going to start from. * The next two lines RUN, will indicate installation commands we want to run. These are the same commands that we used interactively above. * The last line, CMD, indicates the default command we want a container based on this container image to run, if no other command is provided. It is recommended to provide CMD in exec-form. It is written as a list which contains the executable to run as its first element, optionally followed by any arguments as subsequent elements. The list is enclosed in square brackets ([]) and its elements are double-quoted (") strings which are separated by commas. For example, CMD ["ls", "-lF", "--color", "/etc"] would translate to ls -lF --color /etc. The CMD cat /etc/passwd is equivalent to CMD ["/bin/sh", "-c", "cat /etc/passwd"].

An example recipe Dockerfile would be

FROM alphine
RUN apk add --update python3 py3-pip python3-dev
RUN pip install ipython
CMD ["python", "--version"]

Create a new Docker image#

Create a Dockerfile text file.
Use docker image build
- the location of Dockerfile
- the name of new container image like this one USERNAME/CONTAINER_IMAGE_NAME.

docker image build -t USERNAME/CONTAINER_IMAGE_NAME .

The -t option names the container image; the final dot indicates that the Dockerfile is in our current directory.

For example, if my user name was alice and I wanted to call my container image alpine-python, I would use this command: docker image build -t alice/alpine-python .

Notice the input to docker image build is a directory i.e. the current directory ..

Even if it won’t need all of the files in the build context directory, Docker does “load” them before starting to build, which means that it’s a good idea to have only what you need for the container image in a build context directory.